1. Field of the Invention
The present invention relates to an image processing apparatus and method for calculating a parallax for generating a distance image that represents a stereoscopic shape of a subject based on a plurality of images. The invention also relates to a program for causing a computer to perform the image processing method.
2. Description of the Related Art
A method for generating a distance image that represents a stereoscopic shape of a subject is proposed. In the method, the subject is imaged by two or more cameras placed at different positions, then corresponding points, which are pixels corresponding to each other, between a plurality of images (a base image obtained by a base camera and a reference image obtained by a reference camera) obtained by the imaging are searched for (stereo matching) to calculate the positional difference (parallax) between the pixel in the base image and the pixel in the reference image corresponding to each other, and the distance from the base camera or reference camera to the point on the subject corresponding to the pixel is measured by applying the principle of triangulation to the parallax, whereby the distance image is generated.
Here, as illustrated in FIG. 3, when performing the stereo matching, a plurality of points, such as points P1, P2, P3, and so on, is present in real space along a visual line from point O1 which is projected onto pixel Pa in base image G1. Therefore, based on the fact that pixel Pa′ in reference image G2 corresponding to pixel Pa is present on a straight line (epipolar line) representing projected images P1, P2, P3, and so on, the corresponding point which is a pixel in reference image G2 corresponding to pixel Pa is searched for.
In FIG. 3, point O1 is a viewpoint of the camera that obtains base image G1 and point O2 is a viewpoint of the reference camera that obtains reference image G2. Here, the viewpoint is the focal point of optical system of each camera. When performing the stereo matching in the manner as described above, correlation window W that includes pixel Pa, which is a corresponding point search target, is set on base image G1, then correlation window W which is identical to that set on the base image is moved along the epipolar line on reference image G2 to calculate the correlation value between each pixel in the correlation windows W on the respective images at each moved position, and the pixel located in the center of correlation window W in reference image G2 having a correlation value greater than or equal to a predetermined threshold value is obtained as the corresponding point of pixel Pa. Where base image G1 is used as the base, the corresponding point is searched for in reference image G2, and in this case, a corresponding pixel in base image G1 viewed from the pixel in reference image G2 can be regarded as a corresponding point of the pixel in reference image G2.
In the mean time, a distance image is generated for a variety of subjects, so that a plurality of similar objects may sometimes be included in the base image and reference image. Where a plurality of similar objects is included in the images, there may be a case in which a plurality of pixels (here, two pixels), Pb and Pc, is searched out as corresponding points with respect to one pixel Pa in base image G1, as shown in FIG. 5. Where a plurality of corresponding points is searched out in reference image G2 with respect to one pixel in base image G1, it is necessary to determine one of the corresponding points as a valid corresponding point for parallax calculation. Selection of a false corresponding point as the valid corresponding point, however, will result in incorrect parallax calculation.
Consequently, a method in which a corresponding point that minimizes the parallax is selected when a plurality of corresponding points is searched out with respect to one pixel is proposed as described, for example, in U.S. Pat. No. 6,366,691.
The method described in U.S. Pat. No. 6,366,691 allows the selection of a single corresponding point, but the corresponding point that minimizes the parallax is not always the correct corresponding point. Therefore, there may be a case in which the parallax can not be calculated correctly by the method described in U.S. Pat. No. 6,366,691.
The present invention has been developed in view of the circumstances described above, and it is an object of the present invention to enable accurate calculation of parallax when generating a distance image based on the parallax.